-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added s3_storage backend (WIP) #37
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
percona-ysorokin
force-pushed
the
s3_storage_backend
branch
15 times, most recently
from
March 21, 2024 01:03
577e252
to
4dca5ef
Compare
…orkflows Reworked how code is checked out in GitHub Workflows. Inside '${{github.workspace}}' we now have: - 'src' (for the main project source tree) - 'aws-sdk-cpp' (for AWS SDK C++ source tree with submodules) All Github Workflows 'checkout' actions are now instructed to fetch tags as well ('fetch-tags: true'). For our own code we also fetch full commit history ('fetch-depth: 0'). Reworked the way how we form labels for matrix jobs: '<BUILD_TYPE>-<COMPILER_LABEL>', where '<BUILD_TYPE>' is either 'Debug', 'RelWithDebInfo', or 'ASan' and '<COMPILER_LABEL>' is either 'clang17' or 'gcc13'. These labels are used in directory names and in cache keys. Changed the way how Boost libraries are cached: instead of caching the binary tarball ('.tar.bz2'), the unpacked directory is cached. This helps to eliminate compressing already compressed data. Caching key now also includes the library version (currently, 'boost-libraries-1-83-0'). Added a new step that shows '${{github.workspace}}' and '${{runner.temp}}' for diagnostic purposes. Added a new step that shows the content of the following directories: - '${{github.workspace}}' - '${{runner.temp}}' - '${{runner.temp}}/deps' for diagnostic purposes. GitHub Workflows now builds AWS SDK C++ libraries. * Currently the selected version is 1.11.286 but it is configurable via ${{env.AWS_SDK_CPP_MAJOR}}, ${{env.AWS_SDK_CPP_MINOR}}, and ${{env.AWS_SDK_CPP_PATCH}} variables. * The code is checked out from the 'aws/aws-sdk-cpp' GitHub repo (recursively with submodules) by '1.11.286' tag. * CMake is instructed to build only static versions of the libraries with all the dependencies ('-DBUILD_SHARED_LIBS=OFF' and '-DFORCE_SHARED_CRT=OFF'). * The same compiler and build configuration as for the main project are used to build the libraries ('-DCMAKE_BUILD_TYPE=...', '-DCMAKE_C_COMPILER=...', and '-DCMAKE_CXX_COMPILER=...'). * C++ standard is set to 'c++20' as in the main project ('-DCPP_STANDARD=20'). * In order to instruct AWS SDK C++ to use 'libc++' standard library implementation (for 'clang' compiler), '-DCMAKE_CXX_FLAGS_INIT=-stdlib=libc++' option is passed manually for such configurations. * For Address Sanitizer builds '-DENABLE_ADDRESS_SANITIZER=ON' option is added. * Unity build is enabled ('-DENABLE_UNITY_BUILD=ON'). * Building/running tests is disabled ('-DENABLE_TESTING=OFF' and '-DAUTORUN_UNIT_TESTS=OFF'). * Currently only 's3-crt' component is build ('-DBUILD_ONLY=s3-crt'). * The full specification of these options is available at https://docs.aws.amazon.com/sdk-for-cpp/v1/developer-guide/cmake-params.html * 'conan' package manager recipe https://github.com/conan-io/conan-center-index/blob/master/recipes/aws-sdk-cpp/all/conanfile.py and 'vcpkg' port https://github.com/microsoft/vcpkg/blob/master/ports/aws-sdk-cpp/portfile.cmake were taken as examples. The libraries are built in the '${{github.workspace}}/aws-sdk-cpp-build-${{matrix.config.label}}' directory. The libraries are installed into the '${{runner.temp}}/deps/aws-sdk-cpp-install-${{matrix.config.label}}' directory. The build results (the content of the installation directory) are cached under the 'ws-cpp-sdk-libraries-<SDK_VERSION>-<BUILD_TYPE>-<COMPILER_LABEL>' key (e,g, 'aws-cpp-sdk-libraries-1-11-286-RelWithDebInfo-clang17'). This allows to skip building AWS SDK C++ libraries every time a new PR is created. The rebuild is expected to happen only when the SDK version is updated or a new version of the compiler is added. '-DCPP_STANDARD=20' CMake configuration option is now also added for the main project to synchronize with AWS SDK C++ libraries. Main 'CMakeLists.txt' file now tries to find AWS SDK C++ libraries via 'find_package()'. Developers are now expected to pass '-DCMAKE_PREFIX_PATH=<AWS_SDK_CPP_INSTALL_DIR>' CMake option when they configure the project, where '<AWS_SDK_CPP_INSTALL_DIR>' should point to the AWS SDK C++ installation directory. Main application target now depends on 'aws-cpp-sdk-s3-crt'. Added a stub for the second implementation of the 'binsrv::basic_storage_backend' interface, called 'binsrv::s3_storage_backend', which in future will provide basic operations for working with AWS S3 storage. 'binsrv::storage_backend_factory' factory class extended with a new concrete implemetation. Now: - for 'fs' we return an instance of 'binsrv::filesystem_storage_backend' - for 's3' we return an instance of 'binsrv::s3_storage_backend'. 'binsrv::basic_storage_backend' interface extended with one more abstract method 'get_description()': - 'binsrv::filesystem_storage_backend' ('fs') implementation returns "local filesystem" - 'binsrv::s3_storage_backend' ('s3') implementation returns "AWS S3 (SDK <SDK_VERSION>)" ('<SDK_VERSION>' is populated from the 'Aws::SDKOptions' struct initialized via 'Aws::InitAPI()' call). Main application extended with writing 'get_description()' output into the log.
percona-ysorokin
force-pushed
the
s3_storage_backend
branch
from
March 21, 2024 03:02
4dca5ef
to
23d7a72
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reworked how code is checked out in GitHub Workflows.
Inside '${{github.workspace}}' we now have:
All Github Workflows 'checkout' actions are now instructed to fetch tags as well
('fetch-tags: true'). For our own code we also fetch full commit history
('fetch-depth: 0').
Reworked the way how we form labels for matrix jobs:
'<BUILD_TYPE>-<COMPILER_LABEL>', where
'<BUILD_TYPE>' is either 'Debug', 'RelWithDebInfo', or 'ASan' and
'<COMPILER_LABEL>' is either 'clang17' or 'gcc13'.
These labels are used in directory names and in cache keys.
Changed the way how Boost libraries are cached: instead of caching the binary
tarball ('.tar.bz2'), the unpacked directory is cached. This helps to eliminate
compressing already compressed data. Caching key now also includes the library
version (currently, 'boost-libraries-1-83-0').
Added a new step that shows '${{github.workspace}}' and '${{runner.temp}}' for
diagnostic purposes.
Added a new step that shows the content of the following directories:
for diagnostic purposes.
GitHub Workflows now builds AWS SDK C++ libraries.
${{env.AWS_SDK_CPP_MAJOR}}, ${{env.AWS_SDK_CPP_MINOR}},
and ${{env.AWS_SDK_CPP_PATCH}} variables.
with submodules) by '1.11.286' tag.
with all the dependencies ('-DBUILD_SHARED_LIBS=OFF' and
'-DFORCE_SHARED_CRT=OFF').
build the libraries ('-DCMAKE_BUILD_TYPE=...', '-DCMAKE_C_COMPILER=...', and
'-DCMAKE_CXX_COMPILER=...').
implementation (for 'clang' compiler), '-DCMAKE_CXX_FLAGS_INIT=-stdlib=libc++'
option is passed manually for such configurations.
'-DAUTORUN_UNIT_TESTS=OFF').
https://docs.aws.amazon.com/sdk-for-cpp/v1/developer-guide/cmake-params.html
https://github.com/conan-io/conan-center-index/blob/master/recipes/aws-sdk-cpp/all/conanfile.py
and 'vcpkg' port
https://github.com/microsoft/vcpkg/blob/master/ports/aws-sdk-cpp/portfile.cmake
were taken as examples.
The libraries are built in the
'${{github.workspace}}/aws-sdk-cpp-build-${{matrix.config.label}}' directory.
The libraries are installed into the
'${{runner.temp}}/deps/aws-sdk-cpp-install-${{matrix.config.label}}' directory.
The build results (the content of the installation directory) are cached under
the 'ws-cpp-sdk-libraries-<SDK_VERSION>-<BUILD_TYPE>-<COMPILER_LABEL>' key
(e,g, 'aws-cpp-sdk-libraries-1-11-286-RelWithDebInfo-clang17'). This allows to
skip building AWS SDK C++ libraries every time a new PR is created. The rebuild
is expected to happen only when the SDK version is updated or a new version of
the compiler is added.
'-DCPP_STANDARD=20' CMake configuration option is now also added for the main
project to synchronize with AWS SDK C++ libraries.
Main 'CMakeLists.txt' file now tries to find AWS SDK C++ libraries via
'find_package()'. Developers are now expected to pass
'-DCMAKE_PREFIX_PATH=<AWS_SDK_CPP_INSTALL_DIR>' CMake option when they configure
the project, where '<AWS_SDK_CPP_INSTALL_DIR>' should point to the AWS SDK C++
installation directory.
Main application target now depends on 'aws-cpp-sdk-s3-crt'.
Added a stub for the second implementation of the
'binsrv::basic_storage_backend' interface, called 'binsrv::s3_storage_backend',
which in future will provide basic operations for working with AWS S3 storage.
'binsrv::storage_backend_factory' factory class extended with a new concrete
implemetation. Now:
'binsrv::basic_storage_backend' interface extended with one more abstract method
'get_description()':
"local filesystem"
"AWS S3 (SDK <SDK_VERSION>)" ('<SDK_VERSION>' is populated from the
'Aws::SDKOptions' struct initialized via 'Aws::InitAPI()' call).
Main application extended with writing 'get_description()' output into the log.